Guide to Inpatient Quality Indicators: Quality of Care in Hospitals – Volume, Mortality, and Utilization

ors recorded information about each article on a one-page abstraction form. Informationcoded included: • Clinical domain (i.e. medical, surgical, obstetric, pediatric, and psychiatric) • Clinical rationale for the indicators. • Use of data beyond hospital discharge data. Version 2.1B-8Revision 3 (July 21, 2004) • Strengths and weaknesses identified by the author. • Discuss a novel indicator, as opposed to indicators defined elsewhere and used in the articleonly to discuss its relationship with another variable (i.e., socioeconomic status, race,urbanization). • Scoring method (i.e. rate, ratio, mean, proportion).• Strengths and weaknesses not identified by the author. Each abstraction form was reviewed by the research coordinator for quality of the abstraction andfor accuracy of the coding. All data were then entered into a Microsoft Access database. Step 3: Full abstraction. The purpose of the full abstraction phase was to identify potentialindicators for the new QI set, and to assess the evidence for validity of existing indicators. To accomplishthis, only articles that described an indicator in conjunction with specific and comprehensive informationon its validity were fully abstracted. Four of the original abstractors participated in this phase of theabstraction. Three of these abstractors were medical doctors, the fourth a master’s level researchcoordinator. Each of the articles for preliminary abstraction and the corresponding abstraction form wasreviewed by both the research coordinator and the project manager independently. To qualify for fullabstraction, the articles needed to meet the previously noted criteria and the following criteria: • Define a quality indicator, as opposed to only a relationship that was not formulated orexplicitly proposed as a measurement tool. • Define an indicator based on administrative data only. Only 27 articles met these formal criteria. This highlights an important aspect of the literature on qualityindicators: most indicators are based on published clinical literature to identify important patient andprovider characteristics and processes of care for specific clinical conditions; there is also a substantialliterature on technical aspects such as severity adjustment, coding, and data collection. It should be notedthat, while only 27 articles qualified for formal abstraction, these are not the only useful articles. Manyarticles provide important information about quality measurement. However, few quality indicators arespecifically defined, evaluated, and reported in the literature besides descriptive information on theprocess of development. (The Complication Screening Program is a noteworthy and laudable exceptionthat has been extensively validated in the published literature, mostly by the developers). This evidencereport will be an important contribution to the paucity of literature on indicator validation. An abstraction form was filled out for each indicator defined in an article. The abstraction formcoded the following information: • All the information coded in the preliminary abstraction form.• Measure administrative information (i.e. developer, measure set name, year published).• Level of care (primary (prevention), secondary (screening or early detection) or tertiary(treatment to prevent mortality/morbidity)). • A priori suggested quality standard (i.e. accepted benchmark, external comparison, andinternal comparison).• Indicator definition (numerator, denominator statements, inclusions, and exclusions).• Extent of prior use.• Current status (i.e. measure defined, pilot tested, implemented, discontinued).• Scientific support for measure (i.e. published guidelines, clinician panel, literature review,revision of pre-existing instruments, theory only).• Other essential references for the measure.• Validity testing.• Risk adjustment. Version 2.1B-9Revision 3 (July 21, 2004) If the measure included risk adjustment, a separate form for the risk adjustment method was filledout. This included: • Adjustment rationale. • Published performance for discrimination and calibration. • Extent of current use. Parallel Step: Supplementing literature review using other sources. Because the literature inthis area is not the primary source for reporting the use of quality indicators, a list of suitable indicatorswas compiled from a variety of sources. As previously noted, the phone interviews with project advisorsled to information on some indicators. In addition, the Internet sites of known organizations using qualityindicators; the CONQUEST database; National Library of Healthcare Indicators (NLHI), developed by theJoint Commission on Accreditation of Healthcare Organizations (JCAHO); and a list of ORYX-approvedindicators provided by the JCAHO were searched. Indicators that could be defined using administrativedata were recorded in an indicator database. The result of Phase 1 was a list of potential indicators with varied information on each dependingon the source. Since each indicator relates to an area that potentially screens for quality issues, astructured evaluation framework was developed to determine measurement performance. A series ofliterature searches were then conducted to assemble the available scientific evidence on the qualityrelationship each indicator purported to measure. Due to limited resources, not all of the indicatorsidentified in Phase 1 could be reviewed, and therefore some were selected for detailed review using theevaluation framework. The criteria used to select these indicators are described later. Step 1. Development of evaluation framework. As described previously, a structuredevaluation of each indicator was developed and applied to assess indicator performance in six areas:• Method administrative information. • Classification or analytic approach (i.e. stratification, logistic or linear regression)• System development method (i.e. logistic regression, score based on empirical model, apriori/clinical judgement). • Use of comorbidities, severity of illness, or patient demographics.• Use of longitudinal data, or additional data sources beyond discharge data. • Other essential references for the method.• Abstractor comments. The abstraction forms were reviewed by the research coordinator and entered into a Microsoft Accessdatabase. Breakdown of indicators by primary source. During Phase 1, no one literature search wassufficiently sensitive for the purpose of identifying either quality indicators or quality relationships. Inaddition, there was relatively little literature defining quality indicators. Web sites, organizations, andadditional literature describing quality indicators were searched to be confident that a large percentage ofthe quality indicators in use were identified. In general, most volume, utilization, and ACSC indicatorshave been described primarily in the literature. On the other hand, the primary sources for most mortalityand length of stay indicators were current users or databases of indicators. However, many indicatorsfound in the literature were also reported by organizations, and vice versa. Thus, it is difficult to delineatewhich indicators were derived only from the literature and which were derived from the parallel stepdescribed above. Phase 2: Evaluation of Indicators • Face validity• Precision• Minimum bias• Construct validity Version 2.1B-10Revision 3 (July 21, 2004) • Fosters real quality improvement• Prior use Step 2. Identification of the evidence. The literature was searched for evidence in each of thesix areas of indicator performance described above, and in the clinical areas addressed by the indicators.The search strategy used for Phase 2 began with extensive electronic searching of MEDLINE, PsycINFO,and the Cochrane Library. [170-172] (A decision was made not to search EMBASE on the grounds thatthe studies of quality measurement necessarily must take into account the particular health care systeminvolved. [173]) In contrast to conducting systematic reviews of purely clinical topics, it was reasoned thatthe European literature not captured in the Medline database or Cochrane Library would almost certainlyrepresent studies of questionable relevance to the U.S. health system. The extensive electronic search strategy involved combinations of MeSH terms and keywordspertaining to clinical conditions, study methodology, and quality measurement (Figure B-1). Additional literature searches were conducted using specific measure sets as “keywords”. Theseincluded “Maryland Quality Indicators Project,” “HEDIS and low birth weight, or cesarean delivery, orfrequency, or inpatient utilization,” “IMSystem,” “DEMPAQ,” and “Complications Screening Program.” The bibliographies of key articles were searched, and the Tables of